Can you give an example of the syntax (I assume in mods.toml) to declare such an incompatibly?

The mandatory field is replaced by type=optional/required:

 [[dependencies.somemod]]
   modId="othermod"
   versionRange="[1,)"
-  mandatory=false
+  type="optional"

As for the incompatibility declaration:

[[dependencies.neotests]]
type="discouraged" # can be "incompatible"
modId="client_command_test"
versionRange="[1.4.3,)"
reason="We conflict because of reasons"

The version range can be omitted which will declare an incompatibility with all versions of the mod.

eh I'm not really sure if I like the name discouraged

The name is descriptive, it's quite literally what the feature does. It shows a screen discouraging the use of the mod. We've gone through different names and found this pair the best possible name, that also isn't extremely generic like type="warn" or type="error".
(also, the SPI PR with that naming has already been merged, right now FML is waiting)